Interactive System Effectiveness Evaluation

1. Overview The effectiveness of an interactive system can be defined as the degree to which users can access and use the functionality of a system in meeting their task objectives (Nielsen, 1993). Such assessment is generally based on a multitude of measures. These measures involve determining if users can perform tasks supported by a system: (1) efficiently (i.e., the quantity of performance effectiveness); with (2) minimal errors (i.e., the quality of performance effectiveness); (3) minimal support (i.e., intuitiveness in terms of learnability); (4) maximal retention of commands over time (i.e., intuitiveness in terms of memorability); (5) appropriate task demands (i.e., positive subjective perception in terms of perceived mental workload); and (6) sufficient satisfaction (Eberts, 1994; Shneiderman, 1992). While there are advantages and disadvantages to these evaluative criteria, each provides valuable information in the assessment of system effectiveness and, in turn, human performance with a system, thus warranting the use of a combination of indicators. In addition, because of the conflicting nature of these criteria (e.g., task efficiency may need to be sacrificed in order to improve subjective satisfaction) it is not always possible to achieve optimal performance for all criteria (Nielsen, 1993). Subjective criteria (e.g., user satisfaction, mental workload) can be evaluated by expert evaluation, informal observation of users, openor closedended surveys, and informal experimentation. Objective criteria (e.g., performance time, frequency of errors) can be assessed via analytical-predictive modeling, formal observations of users, closed-ended surveys, and formal experimentation (Preece et al., 1994).